Manipulating the Alpha Level Cannot Cure Significance Testing.

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p = 0.05 to p = 0.005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of 0.05, 0.01, 0.005, or anything else, is not acceptable.

Three more steps toward better science.

Science has striven to do better since its inception and has given us good philosophies, methodologies and statistical tools that, in their own way, do reasonably well for purpose. Unfortunately, progress has also been marred by warring among different perspectives, leading to troubles such as the current reproducibility crises. Here I wish to propose that science could do better with more resilient structures, more useful methodological tutorials, and clearer signaling regarding how much we can trust what it produces.

Retract p < 0.005 and propose using JASP, instead.

Seeking to address the lack of research reproducibility in science, including psychology and the life sciences, a pragmatic solution has been raised recently:  to use a stricter p < 0.005 standard for statistical significance when claiming evidence of new discoveries. Notwithstanding its potential impact, the proposal has motivated a large mass of authors to dispute it from different philosophical and methodological angles. This article reflects on the original argument and the consequent counterarguments, and concludes with a simpler and better-suited alternative that the authors of the proposal knew about and, perhaps, should have made from their Jeffresian perspective: to use a Bayes factors analysis in parallel (e.g., via JASP) in order to learn more about frequentist error statistics and about Bayesian prior and posterior beliefs without having to mix inconsistent research philosophies.

Fisher, Neyman-Pearson or NHST? A tutorial for teaching data testing.

Despite frequent calls for the overhaul of null hypothesis significance testing (NHST), this controversial procedure remains ubiquitous in behavioral, social and biomedical teaching and research. Little change seems possible once the procedure becomes well ingrained in the minds and current practice of researchers; thus, the optimal opportunity for such change is at the time the procedure is taught, be this at undergraduate or at postgraduate levels. This paper presents a tutorial for the teaching of data testing procedures, often referred to as hypothesis testing theories. The first procedure introduced is Fisher's approach to data testing-tests of significance; the second is Neyman-Pearson's approach-tests of acceptance; the final procedure is the incongruent combination of the previous two theories into the current approach-NSHT. For those researchers sticking with the latter, two compromise solutions on how to improve NHST conclude the tutorial.

Health benefits of deer and elk velvet antler supplements: a systematic review of randomised controlled studies.

AIMS: The aim of this systematic review was to evaluate the evidence from RCTs of velvet antler supplements for any condition, using the QUOROM statement as a guiding framework. METHODS: Four electronic databases (PubMed, Medline, Web of Science and Academic search premier, via the bibliographical platform, Endnote) and two review articles were searched for all randomised clinical trials of velvet antler supplements. Retrieved trials were evaluated according to standardised criteria. RESULTS: Seven RCTs were identified as satisfying all inclusion criteria and examined the effectiveness of velvet antler for rheumatoid arthritis (2), osteoarthritis (1), sexual function (1), and sporting performance enhancement (3). Their methodological quality ranged from 3-5, as measured on the Jadad scale. Two RCTs reported some positive effects of velvet antler supplements, but neither were convincing while the remaining five RCTs found no effect of velvet antler supplements. CONCLUSIONS: Claims made for velvet antler supplements do not appear to be based upon rigorous research from human trials, although for osteoarthritis the findings may have some promise.